Patient face as touchpad user interface

ABSTRACT

A method of registering a patient face and using it as a touchpad interface is provided. The method includes steps of acquiring a three-dimensional anatomical image of a patient face, identifying multiple anatomical points at the predefined locations on the patient face in a first coordinate system and assigning at least one function to at least one predefined location, receiving multiple positions in a second coordinate system at the respective predefined locations on the patient face, registering the first and second coordinate systems, by correlating between the positions and the respective anatomical points on the patient face, triggering a command corresponding to the at least one assigned function and communicating the command to an electronic device. A system for registering a patient face by the method is described.

SUMMARY

In an aspect, the invention relates to a method for registering apatient face and using it as a touchpad user interface. The methodcomprises acquiring a three-dimensional anatomical image of a patientface and identifying multiple anatomical points corresponding torespective predefined locations on the patient face in a firstcoordinate system and assigning at least one function to at least onepredefined location. The method further comprises receiving multiplepositions in a second coordinate system at the respective predefinedlocations on the patient face and registering the first and secondcoordinate systems, by correlating between the positions and therespective anatomical points on the patient face. The method alsocomprises triggering a command corresponding to the at least oneassigned function, and communicating the command to an electronicdevice.

In an aspect, the invention relates to a system for registering apatient face and using it as a touchpad user interface. The systemcomprises: a registration tool, which comprises a position sensor of aposition-tracking system, which is configured to acquire multiplepositions in a second coordination system by positioning theregistration tool at respective predefined locations on a patient face;and a processor. The processor is configured to identify, in athree-dimensional anatomical image of the patient face, multipleanatomical points corresponding to the respective predefined locationsin a first coordinate system and to assign at least one function to atleast one predefined location on the patient face. The processor is alsoconfigured to receive the multiple positions measured in the secondcoordinate system, and register the first and second coordinate systems,by correlating between the positions and the respective anatomicalpoints on the patient face. The processor is also configured to retrievea command corresponding to the at least one assigned function.

BRIEF DESCRIPTION OF THE DRAWINGS

The following detailed description of the embodiments of the presentinvention will be better understood when read in conjunction with theappended drawings. For the purpose of illustration, there are shown inthe drawings embodiments which are presently preferred. It isunderstood, however, that the invention is not limited to the precisearrangements and instrumentalities shown. In the drawings:

FIG. 1 is a schematic drawing of a surgical system, in accordance withan embodiment of the present invention.

FIG. 2 is a flow chart that schematically illustrates a method forregistering a patient face with a coordinate system of magnetic positiontracking system and using the patient face as a user interface tocontrol equipment, in accordance with an embodiment of the presentinvention.

FIG. 3 is a schematic drawing of the three-dimensional computertomography image of the patient face used in the registration methodillustrated in FIG. 2.

FIG. 4 is a schematic drawing of the patient face used as a userinterface to control equipment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present disclosure relates generally to methods of utilizing apatient's face as a user interface to facilitate usability ofregistration equipment.

As used herein, the term “registration” refers to a process fordetermining the spatial relationship between body parts and athree-dimensional (3D) computer image constructed, for example, from thepreoperative computer tomography (CT) scan. Surgical navigation systemsare used to guide the physician movements during medical procedures.

In certain medical procedures, such as Ear-Nose-Throat (ENT) surgery, ananatomical image of relevant organs may be registered with a coordinatesystem of a position tracking system. Using the registration, a surgicaltool fitted with a position sensor may be navigated to the treatedorgans, and can be visualized overlaid on the anatomical image. Inprinciple, pre-operative registration may be carried out using anexternal registration tool fitted with a position sensor of the positiontracking system. Such a tool could be applied to the preselectedlocations on the patient face (e.g., forehead, and centers of the twocheeks). The anatomical image could then be registered to the coordinatesystem of the position tracking system based on the measured positionsof tissue at the preselected locations.

Computerized tomography imaging systems and position-tracking systemsare known in the art. Methods to register medical images are described,for example, in U.S. Pat. No. 6,560,354 and U.S. patent application Ser.No. 15/674,380, which are incorporated herein by reference as if fullyset forth. U.S. Pat. No. 6,560,354 describes apparatus and method forregistration of images to physical space using a weighted combination ofpoints and surfaces. An image of a patient taken through X-ray computedtomography is registered to physical measurements taken on the patient'sbody. Different parts of the patient's body are given differentnumerical weights; for example, if bone measurements are deemed to bemore accurate than skin measurements, the bones can be given a higherweight than the skin. The weights are used in an iterative registrationprocess to determine a rigid body transformation function.

U.S. patent application Ser. No. 15/674,380 describes a method and anapparatus for Ear-Nose-Throat (ENT) registration. The method includesreceiving a computerized tomography (CT) image of voxels of a subject'shead, and analyzing the image to identify respective locations of thesubject's eyes in the image, so defining a first line segment joiningthe respective locations. The method also includes identifying a voxelsubset overlaying bony sections of the head, lying on a second linesegment parallel to the first line segment and on a third line segmentorthogonal to the first line segment. A magnetic tracking systemconfigured to measure positions on the subject's head is activated, anda probe, operative in the system is positioned in proximity to the bonysections to measure positions of a surface of the head overlaying thebony sections. A correspondence between the positions and the voxelsubset is formed, and a registration between the CT image and themagnetic tracking system is generated in response to the correspondence.

Physicians, such as Ear-Nose-Throat (ENT) physicians, often performmedical procedures without the assistance of a nurse. For example, inthe case of ENT mapping and registration procedures, the ENT physicianperforms procedures in sterile environments, and is himself/herselfscrubbed and sterilized. Thus, it is inconvenient for the physician toremove gloves to perform the registration and touch a mouse or keyboardduring the procedure. However, the physician still needs to use thecomputer and is often alone with the equipment. Therefore, physiciansneed an improved system that facilities computer use during procedureswhen nurses are not present in the room.

The proposed system utilizes the patient's face as a touchpad userinterface. For example, during an ENT registration procedure, thephysician (who is scrubbed and sterilized) will not need to removegloves to operate a mouse or keyboard. Instead, the physician can usethe patient's face and other parts of the head as a touchpad interface.

System Description

FIG. 1 is a schematic drawing of a surgical system 100, in accordancewith an embodiment of the present invention. System 100 comprises amagnetic position tracking system, which is configured to track theposition of one or more position sensors (not shown) in the head of apatient 101. The magnetic position tracking system comprises magneticfield generators 104 and one or more position sensors. The positionsensors generate position signals in response to the sensed externalmagnetic fields from the field generators and enable a processing device102 to map the position of each sensor in the coordinate system of theposition tracking system as described herein.

The methods of position system is known in the art and described in U.S.Pat. Nos. 5,391,199; 6,690,963; 6,484,118; 6,239,724; 6,618,612 and6,632,089; in International Patent Publication No. WO 96/05768; and inU.S. Patent Application Publications 2002/0065455; 2003/0120150 and2004/0068178, all of which are incorporated herein by reference as iffully set forth. Position system 100 comprises a location pad 103, whichcomprises multiple field generators 104 fixed to a frame 105. Referringto FIG. 1, pad 103 comprises five field generators 104, but any othersuitable number of field generators 104 can be used. Pad 103 comprises apillow 106 placed under a head of patient 101, such that generators 104are located at fixed known positions external to the patient. System 100further comprises a console 108, which comprises a driver circuit (notshown) configured to drive field generators 104 with suitable signal soas to generate magnetic fields in a predefined working volume aroundhead 107.

In an embodiment, system 100 comprises a registration tool. Theregistration tool 112 may be a handheld wand. Wand 112 may be used bysystem 100 for registering the coordinate system of the magnetictracking system with that of pre-acquired computerized tomography (CT)image. The registration tool 112 is configured to acquire positionmeasurement.

In an embodiment, processor 102 is a general-purpose computer comprisingsuitable front and interface circuits for receiving data from externalsources, as well as measurements from the position sensor of wand 112via cable 109, and for controlling other components of system 100.Console 108 further comprises input devices 113 and a screen display114, which is configured to display the data.

A physician 110 may place wand 112 to a predefined location on patienthead 107. Each predefined location is a portion of face 115. Thepredefined location may be any one of quarters or quadrants of the face:upper-right quadrant, upper-left quadrant, lower-right quadrant, orlower-left quadrant, or any other suitable identifiable portion.Exemplary predefined locations of patient face 115 are illustrated inFIG. 4.

In an embodiment, processing device 102 receives a computerizedtomography (CT) image 111 obtained using an external CT system (notshown).

Processing device 102 may use image 111 to project a surface image ofpatient face 115 on input device 113 or to screen display 114 to assistthe physician and/or serve as a user interface. Processing device 102may distinguish different portions of the CT image 111. Processingdevice 102 may identify a nose, or eyes, using any suitable criterion ortechnique such as hounsfield unit (HU). Processing device 102 mayanalyze the CT image 111 to identify and may define different portionsof the face. The different portions of the face may be quadrants 201,202, 203 and 204 as shown in FIG. 3.

In an embodiment, when placed at a predefined location on the patientface, wand 112 is configured to generate position signals indicative ofthis predefined location in the coordinated system of the magneticposition tracking system. The processing device 102 stores positionsignals or points acquired in predefined locations in each of thequadrants 201, 202, 203 and 204. The minimum number of points may berespectively 20, 20, 12, and 12. However, the minimum number of pointsmay be lesser or greater than any one of these values, and may be thesame or different value for all quadrants.

In an embodiment, processing device 102 is configured to calculate twocoordinates for each predefined location on the patient head-“ananatomical point” in a coordinate system of the CT system, and a“position” in a coordinate system of the position tracking system. Theanatomical point is derived from the position measurement of wand 112 atthe predefined location, and is indicative of the coordinate of theportion of the face at this location in the coordinate system of themagnetic position tracking system. The anatomical point is indicative ofthe coordinate of the portion of the face, as identified in the CT image111.

In an embodiment, processing device 102 is configured to correlate theanatomical points and the positions of the predefined locations in image111, so as to register the CT image with the coordinate system of theposition tracking system.

FIG. 2 is a flow chart of the steps of an exemplary method forregistering the patient's face with a coordinate system of magnetictracking system as a user interface to control equipment according toembodiments disclosed herein. FIG. 3 is a schematic drawing of the 3D CTimage of the patient face used in the registration method illustrated inFIG. 2. As illustrated in FIG. 2, the method may comprise the step S1 ofacquiring an image of the patient face and/or displaying the image on adisplay screen. The image may be a three-dimensional (3D) ultrasoundimage, a CT image, MRI image or an image obtained by any other imagingtechnique known in the art.

In step S2, processing device 102 analyzes the 3D CT image of thepatient face 111 as shown in FIG. 3 using image processing software andfinds a highest point 192 on the image. Typically, the highest pointcorresponds to the nose tip of the patient.

In step S3, processing device 102 finds centers 190 and 191 of the eyesof the patient as illustrated in FIG. 3. Methods for finding centers ofthe eyes are known in the art and are described, for example, in thearticle entitled “Locating the Eyes of the Patient” published in theProceedings of the 6th International Conference on Industrial andEngineering Applications of Artificial Intelligence and Expert Systems,1993, pages 507-517, which is incorporated herein by reference as iffully set forth. Other methods for finding the centers of the eyes mayalso be used.

In step S4, processing device 102 may analyze surface of the 3D image111 using values acquired in steps S2 and S3 using the image processingsoftware to delineate voxels within the cross-shape (

) in the acquired image. The cross-shape comprises the bridge of thepatient's nose as a vertical line 194 (FIG. 3) of the cross (

). To determine voxels of the vertical line the processor starts fromthe nose tip (highest point) determined in step S2, and finds othervoxels of the bridge of the nose, i.e., the bony section of the nose, bylooking for local maxima in surface image 111 in proximity to, andvertically above, the nose tip 192. The processor continues this processiteratively to find voxels corresponding to the complete vertical line194 of the cross-shape. To find voxels corresponding to a horizontalline 193 of the cross-shape, the processor selects voxels that are apreset vertical distance above a line joining the patient's eye centers,as found in step S3. In one embodiment, the preset distance is 5 cmabove the eye center line, but in other embodiments the distance may belarger or smaller than 5 cm. The preset distance is chosen so that thevoxels of the horizontal line overlie the bony section of the patient'sforehead. The processing device 102 draws line 193 above the line thatjoins the two eye centers found in step S3. The processing device drawsa line 194 orthogonal to line 193 and passing through the highest pointon the image of patient face. Typically, both the vertical line and thehorizontal line of voxels are more than one voxel wide. As shown in FIG.3, the two lines divide the image of the face into four quadrants 201(upper-right quadrant), 202 (upper-left quadrant), 203 (lower-rightquadrant) and 204 (lower-left quadrant).

In an image analysis step S4, the processing device 102 may analyze 3Dimage 111 generated in step Si, using the values acquired in steps S2and S3. The processing device may generate four subsets of the set ofvoxels comprising 3D image 111. Each one of the subsets corresponds torespective quadrant of four quadrants 201 (upper-right quadrant), 202(upper-left quadrant), 203 (lower-right quadrant) and 204 (lower-leftquadrant).

In step S5, processing device 102 may define commands and assignfunctions to each quadrant on the patient face as shown in FIG. 4. Thecommands may be any commands used during medical procedure. The commandsmay be commands used during registration or related to registration. Forexample, touching the lower-right quadrant on the patient's face may beassociated with the “Yes” function during the process of registration,while touching the lower-left quadrant may be associated with the “No”function during the registration. The physician may touch the patient'sforehead to “Re-Do” the registration. The assigned functions may bestored in a database and may be implemented in the form of logicalinstructions or commands that are executed by the processing device,such as a computer. The database may be in the form of a registrationtable. Other commands may also be used. For example, a generic command“Menu” can be used to open additional commands. “Undo” can be used toreverse previous action. “Show”/“Hide” can be used to display or hidecontent, e.g., images. The commands may be commands for zooming in the3D CT image, exiting the application or printing the screen.

In step S6, physician 110 places may place wand 112 comprising aposition sensor at the predefined locations in each of the quadrants ofthe patient face to acquire the positions in the coordinate system ofthe position tracking system. In an embodiment, the predefined locationsmay comprise four points as follows: a point below the tip of thepatient's nose, the left and right sides of the patient's face next tothe eyes and a point between the eyes. However, fewer than four points,e.g., two points, may be sufficient to generate registration.

In step S7, processing device 102 may correlate the points acquired instep S6 with the subsets of voxels generated in step S4 using theItirated Closest Point (ICP) algorithm. The algorithm uses the subsetsof voxels as the reference set of points, and the positions acquired instep S6 as the source set of points. While performing the correlationthe processing device 102 may register between the coordinate systems ofthe CT and the position tracking systems. The processing device 102 maycheck if the registration performed in step S7 is sufficiently accurate,i.e., if the errors associated with the cost function generated by theICP algorithm are sufficiently small, for example, below a presetthreshold. If the condition returns positive, then a notice is providedon screen 114 informing the physician 110 that she/he may ceaseacquiring points. The notice may also suggest that the physician 110performs a verification of the registration, such as by touchingpredefined positions and having the processor mark these positions onthe CT image, and/or by measuring distances between such positions. Ifthe condition returns negative, or if the verification fails, thephysician continues to acquire points with wand 112.

In step S8, the physician 110 may trigger the command by applying a toolequipped with the position sensor in any one of the quadrants of thepatient face. The tool may be positioned in the same place of thequadrant for a defined period of time to trigger the command, forexample, 2-3 seconds. The tool may have the capability to trigger thecommand by clicking the position sensor. In response to trigger, thecommand or function may be communicated to the processing device, whichmay be a computer.

FIG. 4 is a schematic drawing of the patient face used as a userinterface to control equipment in one embodiment. As seen in FIG. 4, thepatient's face can be divided into virtual areas, such as quadrants,where a particular virtual area or quadrant is assigned a specificuser-defined parameter, or function. Each function corresponds to acommand which can be retrieved and performed by the processing device102.

For example, to indicate whether or not the physician accepts theproposed registration, the physician may input a “Yes” by touching thepatient's lower-right quadrant 203, or a “No” by touching the patient'slower-left quadrant 204. The physician may press either the upper-rightquadrant 201 or the upper left quadrant 202 of the patient's forehead to“Re-Do” the registration. In an embodiment, processing device 102 isconfigured to identify each quadrant of the patient face, whichcorresponds to the specifically assigned function.

In an embodiment, the patient face can be divided into as many virtualsections as necessary to accommodate the physician's needs and toprovide functionality as a user interface.

The registration process is performed before the surgical or any othermedical procedure performed on the patient. The registration process istypically performed in a non-sterile environment but may also beperformed in a sterile environment. The medical procedure may beperformed in a sterile environment but with different levels ofsterility depending on the setting, e.g., an operating room, office orclinic. During the medical procedure, physician 110 may touch patientface 115 with a medical tool 112 or surgical device, which comprises anadditional position sensor of the position tracking system. Since the CTimage is already registered with the position-tracking system, physician110 may apply a medical tool 112 to a pre-defined location on patientface 115 to perform certain actions with a computer. The medical toolmay be the wand used for face registration, or any other tool usedduring the medical procedure. The medical tool may be sterile ornon-sterile depending on the type of procedure or setting where the toolis used.

In an embodiment, processing device 102 is configured to receive one ormore images using suitable imaging techniques, and to register theseanatomical images with the coordinate system as described herein. Forexample, suitable imaging techniques may be ultrasound, fluoroscopy ormagnetic resonance imaging (MRI).

FIG. 1 shows only elements related to the disclosed techniques forclarity and simplicity. System 100 may comprise additional modules andelements that are not directly related to the disclosed techniques, andtherefore, these are intentionally omitted from FIG. 1 and from thedescription thereof.

Processing device 102 may also control other components of the system100 according to the embodiments described herein. Processing device 102is preferably programmed in software and/or hardware to perform thefunctions required by the system. The processing device 102 may storedata for the software in a memory. The software may be downloaded to theprocessing device 102 in electronic form, over a network, or may beprovided on tangible media, such as optical, magnetic or othernonvolatile memory media. Alternatively, some or all of the functions ofprocessing device 102 may be performed by dedicated or programmabledigital hardware components.

Accordingly, systems and methods have been described that enableutilizing a patient's face as a user interface to facilitate usabilityof registration equipment. Thus, the proposed system facilitates theusability of equipment during specialized procedures, such as ENTregistration procedures, so that the physician can operate the equipmentwithout removing gloves and without assistance from a nurse.

The proposed system facilitates the usability of specialized equipmentused in a sterile environment for various medical procedures, such asENT registrations, when no nurse is present during the procedure. Thephysician can utilize the patient's face as a user interface to operatethe equipment, which enables the physician to overcome the absence of anurse during the procedure.

It should be understood that many variations are possible based on thedisclosure herein. Although features and elements are described above inparticular combinations, each feature or element can be used alonewithout the other features and elements or in various combinations withor without other features and elements.

The methods provided include implementation in a general purposecomputer, a processor, or a processor core. Suitable processors include,by way of example, a general purpose processor, a special purposeprocessor, a conventional processor, a digital signal processor (DSP), aplurality of microprocessors, one or more microprocessors in associationwith a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs)circuits, any other type of integrated circuit (IC), and/or a statemachine. Such processors can be manufactured by configuring amanufacturing process using the results of processed hardwaredescription language (HDL) instructions and other intermediary dataincluding netlists (such instructions capable of being stored on acomputer readable media). The results of such processing can bemaskworks that are then used in a semiconductor manufacturing process tomanufacture a processor which implements the methods described herein.

The methods or flow charts provided herein can be implemented in acomputer program, software, or firmware incorporated in a non-transitorycomputer-readable storage medium for execution by a general purposecomputer or a processor. Examples of non-transitory computer-readablestorage mediums include a ROM, a random access memory (RAM), a register,cache memory, semiconductor memory devices, magnetic media such asinternal hard disks and removable disks, magneto-optical media, andoptical media such as CD-ROM disks, and digital versatile disks (DVDs).

Certain terminology is used in the description herein for convenienceonly and is not limiting. The words “right,” “left,” “top,” and “bottom”designate directions in the drawings to which reference is made. Thewords “a” and “one,” as used in the claims and in the correspondingportions of the specification, are defined as including one or more ofthe referenced item unless specifically stated otherwise. Thisterminology includes the words above specifically mentioned, derivativesthereof, and words of similar import. The phrase “at least one” followedby a list of two or more items, such as “A, B, or C,” means anyindividual one of A, B or C as well as any combination thereof.

Further embodiments herein may be formed by supplementing an embodimentwith one or more element from any one or more other embodiment herein,and/or substituting one or more element from one embodiment with one ormore element from one or more other embodiment herein.

It is understood, therefore, that this invention is not limited to theparticular embodiments disclosed, but is intended to cover allmodifications which are within the spirit and scope of the invention asdefined by the appended claims; the above description; and/or shown inthe attached drawings.

What is claimed is:
 1. A method, comprising: acquiring athree-dimensional anatomical image of a patient face and identifyingmultiple anatomical points corresponding to respective predefinedlocations on the patient face in a first coordinate system; assigning atleast one function to at least one predefined location on the patientface; receiving multiple positions in a second coordinate system at therespective predefined locations on the patient face; registering thefirst and second coordinate systems, by correlating between thepositions and the respective anatomical points on the patient face;triggering a command corresponding to the at least one assignedfunction; and communicating the command to an electronic device.
 2. Themethod of claim 1, wherein the three-dimensional anatomical imagecomprises a computerized tomography image.
 3. The method of claim 1,wherein the step of identifying further comprises determining thepredefined locations at the three-dimensional anatomical image.
 4. Themethod of claim 3, wherein the step of determining comprises finding thehighest point at the three-dimensional anatomical image.
 5. The step ofclaim 4, wherein the step of determining further comprises finding acenter in each of the right and left eyes at the three-dimensionalanatomical image.
 6. The method of claim 5, wherein the step ofdetermining further comprises dividing the three-dimensional anatomicalimage of patient face into quadrants by two orthogonal lines, whereinthe first orthogonal line is above and parallel to a line that connectsthe center of the right eye and the center of the left eye and thesecond orthogonal crosses the highest point of the three-dimensionalimage.
 7. The method of claim 1, wherein the multiple anatomical pointscomprise at least two points at each of the predefined locations.
 8. Themethod of claim 1, wherein the at least one predefined location isselected from the group consisting of: upper-left quadrant, upper-rightquadrant, lower-left quadrant and lower-right quadrant of the patientface.
 9. The method of to claim 1, wherein receiving the multiplepositions comprises receiving the positions from a registration toolcomprising the position sensor.
 10. The method of claim 1, whereinreceiving the multiple positions comprises receiving the positions fromscanning the patient face with a three-dimensional scanner
 11. Themethod of claim 1, wherein triggering the command comprises touching theat least one predefined location on the patient face with a surgicaltool comprising a position sensor.
 12. The method of claim 1, whereinthe electronic device is a computer.
 13. A system, comprising: aregistration tool, which comprises a position sensor of aposition-tracking system, which is configured to acquire multiplepositions in a second coordination system by positioning theregistration tool at respective predefined locations on a patient face;and a processor, which is configured to: identify, in athree-dimensional anatomical image of the patient face, multipleanatomical points corresponding to the respective predefined locationsin a first coordinate system; assign at least one function to at leastone predefined location on the patient face; receive the multiplepositions measured in the second coordinate system; register the firstand second coordinate systems, by correlating between the positions andthe respective anatomical points on the patient face; and retrieve acommand corresponding to the at least one assigned functions.
 14. Thesystem of claim 13, wherein the three-dimensional anatomical imagecomprises a computerized tomography image.
 15. The system of claim 13,wherein the three-dimensional anatomical image comprises athree-dimensional scan.
 16. The system of claim 13, wherein theprocessor is further configured to determine the predefined locations atthe three-dimensional anatomical image, wherein the step of determiningcomprises: finding the highest point at the three-dimensional anatomicalimage; finding a center in each of the right and left eyes at thethree-dimensional anatomical image; and dividing the three-dimensionalanatomical image of patient face into quadrants by two orthogonal lines,wherein the first orthogonal line is above ad parallel to a line thatconnects the center of the right eye and the center of the left eye andthe second orthogonal crosses the highest point of the three-dimensionalimage.
 17. The system of claim 13, wherein the predefined location isselected from the group consisting of selected from the group consistingof: upper-left quadrant, upper-right quadrant, lower-left quadrant andlower-right quadrant of the patient face.